1. Technical Field The present invention relates to video processing and more particularly, to applying tonal stabilization to video sequences.
2. Discussion of the Related Art
With the proliferation of inexpensive video capturing devices, and the increasing popularity of video sharing websites over the last few years, there has been a dramatic increase in the amount of captured video content. Most of this video footage is home-made and captured by amateur videographers using low-end video cameras.
While professional videographers may employ an elaborate setup to control the motion of the camera and the lighting of the scene, home-made video footage often suffers from camera shake and from significant fluctuations in exposure and color balance. These tonal fluctuations are induced by the camera's automatic exposure and white balance control: minute adjustments to these tonal settings are continuously made in response to changes in the illumination and the composition of the frame. Turning auto-exposure off is not a practical option, since the dynamic range of the scene is typically much greater than what the camera is able to capture with a fixed exposure setting, making it difficult to avoid over- and under-exposure. Turning off automatic white balance is more feasible, but not all cameras offer this option.
While video motion stabilization (elimination of camera shake effects) has been the subject of much research, elimination of tonal fluctuation, or tonal stabilization, got surprisingly little attention.
The digital video capture pipeline may be modeled as follows: the analog linear RGB values arriving at the camera's sensor are converted to digital values, undergo luma/chroma separation, processed to adjust brightness and color, and finally encoded to the target digital video format. Both the analog-to-digital conversion and the subsequent processing may involve non-linear operations. It is customary to refer to the combined effect of this pipeline as the camera's response function, which may vary between different cameras operating at different settings, and is typically proprietary. Had the camera response at each frame been known, it would be possible to stabilize the sequence by inverting the response function.
Some methods are known in the art for modeling and recovering the camera response, including parametric, semi parametric and non-parametric approaches. However, these methods typically operate on still, geometrically registered images, which vary only in their exposure. To apply them to video would require a sufficiently large set of exact correspondences between each pair of frames, which might be difficult to compute. Even if the required correspondences are available, the exposure change between successive frames is typically too small to produce a numerically stable result. Furthermore, it would be necessary to extend these methods to handle more general changes of the camera parameters.
At a first glance, it might seem that tonal alignment may be achieved simply by transferring color from a frame taken from a tonally stable section in the video sequence (hereinafter: an anchor frame) to the remaining frames. Indeed, a variety of color transfer methods have been proposed over the years. Some solutions known in the art proposed matching various global color statistics of two images, such as mean and variance in some color space. However, such methods cannot be used for tonal stabilization, since the statistics of a frame tend to vary significantly due to camera and object motion. These changes can occur quite quickly, and therefore any attempts to match the global statistics would result in introducing fluctuations of their own. Local methods, try to find a local match between regions in the image and fit a corresponding offset. While such transfer models are powerful, reliably matching regions in the presence of camera and scene motion remains a challenging task. Yet another significant problem in using both global and local methods in the context of frame-to-frame color transfer is that of error accumulation.